Skip to main content

RSMS API results — implementation plan (Parquet / DuckDB migration era)

Archived. Checklist referred to migrating reads away from DuckDB toward Azure Table Storage + blobs — that migration is finished in production code. app/results/ plus OpenAPI /docs are the behavioral reference. rsms-master-tasklist captures open follow-ups.

Historical context: duckdb and RSMS-results Parquet reads were retired in favor of Azure Table Storage + blob JSON. The checklist sections below recorded the migration era; app/results/, ssot/rsms-ssot.json, ssot/rsms-domain-units-ssot.md, and route mapping sibling rsms-api-results-duckdb-design (also archived).

1. Scope (v1)​

In scopeOut of scope (v1)
GET .../cxplt/by-hour ( PlumeByTime row)Legacy chart 2 / 5 full CTPLT parity — product backlog
GET .../cxplt/by-mile ( PlumeByMile row)Ad-hoc client queries; claiming legacy chart 2 or 5 semantics from CXPLT-style slices alone
GET .../mass-balance ( mass_balance.json blob)Caching layer beyond minimal (optional later)
GET .../edge-profile, GET .../edge-timeseries (JSON)River-mile backlog — CXPLT dual-axis wired in product when plume_river_mile_axis returns data (rsms-master-tasklist)

2. Dependencies and configuration​

2.1 Python (rsms-api-fastapi)​

  • azure-data-tables and azure-storage-blob (JSON blobs). duckdb is not used for RSMS scenario results.
  • Reuse connection string / managed identity patterns from app/azure_storage.py where applicable.

2.2 Settings (app/config.py / env)​

VariablePurpose
AZURE_STORAGE_CONNECTION_STRINGBlob + Table (same account); used by results for JSON blobs and plume Table rows
RSMS_RESULTS_BLOB_CONTAINERContainer for edge_timeseries.json, mass_balance.json (default rsms-output; align with writer pipeline and SSOT configKeys)

Document in .env.example with comments; no secrets committed.

3. Module layout (implemented)​

rsms-api-fastapi/app/
  results/
    __init__.py
    router.py              # prefix /riverbasins/{id}/scenarios/{id}/results
    constants.py           # Table names (PlumeByTime / PlumeByMile)
    keys.py                  # Partition/Row key helpers
    decode.py                # float32 buffers → series
    plume_index.py           # hour/mile → table indices
    models.py                # Pydantic responses + meta
    service.py               # Table get_entity + blob download + orchestration

main.py: app.include_router(results_router, ...) behind RESULTS_API_ENABLED.

4. Implementation phases​

Phase A — Skeleton + parity with SSOT​

  1. Load ssot/rsms-ssot.json at startup (or embed version string constant synced manually for v1).
  2. Add results/router.py with three routes returning stub JSON (empty series) to validate paths and OpenAPI.
  3. Add Pydantic models matching apiResponses section of SSOT.

Exit: GET routes visible in /docs under correct prefix.

Phase B — Table + blob resolution​

  1. table_source.py / blob_source.py: Given riverbasin_id, scenario_id, build Table partition/row keys (SSOT) and blob paths for mass_balance.json / edge_timeseries.json.
  2. Read path: Single-row get_entity for PlumeByTime / PlumeByMile; download JSON blobs to memory or temp file as needed.
  3. 404 if entity/blob missing (or GET /scenarios/{id} fails first).

Exit: Unit tests with Azurite, in-memory fakes, or checked-in tiny float32 buffers (no production secrets in CI).

Phase C — Decode + JSON assembly​

  1. plume_index.py / service.py: Map hour → time_index, miles → mile_index with SSOT rounding; get_entity on PlumeByTime / PlumeByMile with bound keys.
  2. Binary payloads: numpy.frombuffer(..., dtype=float32); validate length M or T from metadata.
  3. Mass balance: parse mass_balance.json; compute total_mass in Python as pmass + dzmass per SSOT derived.

Exit: Integration test: fixture buffers/JSON → JSON series / rows length > 0.

Phase D — Mass balance + scenario merge​

  1. GET /scenarios/{scenario_id} (existing) — fetch scenario for spill_quantity / quantity field when available (pounds per domain SSOT).
  2. Embed spill_quantity_lb in mass-balance response (null if absent).

Exit: Response matches apiResponses.massBalance in SSOT.

Phase E — Async / performance​

  1. Run Table/blob I/O (sync clients) in asyncio.to_thread() (or run_in_executor) so event loop is not blocked.
  2. Optional: request-level timeout (e.g. 30s).

Exit: Load test note only; no premature optimization.

Phase F — Edge endpoint (optional)​

  1. GET .../edge-profile → 501 + JSON {"reason": "CTPLT not available", "deferred": true} per API design §4.4.

Exit: Frontend can branch without throwing.

Phase G — Documentation and SSOT sync​

  1. Link docs/archive/rsms-api-results-implementation-plan.md from rsms-api-fastapi/README.md if present.
  2. On each behavioral change, update ssot/rsms-ssot.json and bump meta.schemaVersion.

5. Security and validation​

  • UUID validation for path parameters (FastAPI UUID type).
  • Authorization: reuse get_current_user / scenario access rules if/WHEN scenarios are tenant-scoped (mirror GET /scenarios/{id} behavior).
  • No user-supplied storage keys; validated UUIDs and fixed endpoint maps only.

6. Testing strategy​

LayerApproach
Keys / decodePartition keys, index mapping, float32 buffer lengths.
ServiceTable row fixtures or JSON blobs under tests/fixtures/results/.
E2EOptional: Azurite or test storage account.

7. Rollback​

  • Feature flag RESULTS_API_ENABLED (env) to disable router registration if needed.

8. References​

DocRole
rsms-api-results-duckdb-designArchived route ↔ chart mapping (hist.)
rsms-results-table-storage-rework-planActive deploy / ops phased plan
ssot/rsms-ssot.jsonColumns, API shapes
ssot/rsms-domain-units-ssot.mdUnits and domain terms

Implementation plan — RSMS API results reads.

Last updated on by jacobgqc